The present invention comprises an improved method of block based motion compensation in a video image upconversion unit of the type that uses block based motion compensation to generate an interpolated video field using motion vectors. The improved method of the present invention obtains accurate values for motion compensated pixels when incorrect motion vectors are assigned to the pixels.
A moving video image is transmitted by a video transmitter as a sequence of frames or pictures. Each frame or picture may be coded individually, but is displayed sequentially at a video rate. Each video frame is made up of two video fields, an odd video field and an even video field. More specifically, an individual frame denoted by the letter xe2x80x9cAxe2x80x9d is made up of an odd field denoted by the letters xe2x80x9cAoxe2x80x9d and an even field denoted by the letters xe2x80x9cAe.xe2x80x9d
When capturing or recording a video sequence, either frames or fields may be captured. When both the odd field and the even field of a video frame are captured at the same time, the picture is said to be a xe2x80x9cprogressivexe2x80x9d picture. The odd fields and the even fields are not generally used to describe progressive pictures. Instead the individual video frames (e.g., frame A, frame B) are used to describe progressive pictures. Most movie film material is composed of progressive pictures.
When the odd field and the even field of a video frame are captured at different times, the picture is said to be an xe2x80x9cinterlacedxe2x80x9d picture. The two fields are not combined to be displayed at the same time. Each field is treated and displayed separately. Most television material is composed of interlaced pictures.
In order to achieve maximum efficiency during the transmission process, not every field or frame is transmitted. That is, some of the individual fields or frames are dropped and are not transmitted. The dropped fields or frames are recreated by the video receiver from information taken from the fields or frames that are transmitted.
For example, the dropped fields or frames can be recreated simply by repeating the previous field or frame. Alternatively, if the display is delayed, the next field or frame that follows the dropped field or frame can be used to take the place of the dropped field or frame. It is also possible to replace the dropped field or frame by averaging the neighboring fields or frames on each side of the dropped field or frame.
There are difficulties with these simple approaches. Repeating a previous field or frame (or using the next following field or frame) in place of the dropped field or frame causes the perceived image to be jerky even when small motions are depicted in the video image. Averaging the fields or frames causes the perceived image to be blurred even when moderate motions are depicted in the video image.
A well known method for recreating the dropped fields or frames is motion compensated interpolation. In motion compensated interpolation (also referred to as xe2x80x9cbidirectional predictionxe2x80x9d) a subsignal with a low temporal resolution (typically one half to one third of the frame rate) is coded and the full resolution signal is obtained by interpolation of the low resolution signal and the addition of a correction term. The signal to be reconstructed by interpolation is obtained by adding a correction term to a combination of a past and a future reference.
Video sequences exist in various formats. For example, high definition (HD) television video sequences may be displayed in any one of eighteen (18) different formats. The process of converting a video sequence from one format to another format is called xe2x80x9cscan rate conversion.xe2x80x9d
Scan rate conversion may be used to reduce image flicker in televison images. For example, a European televison video standard specifies a frequency of fifty Hertz (50 Hz). That is, the video fields are to be displayed at a rate of fifty fields per second. This televison video rate is not sufficient to prevent noticeable flicker in the television image. To reduce image flicker, the television video rate may be increased to one hundred Hertz (100 Hz) by interpolating additional fields between the original fields in the video image.
Scan rate conversion techniques may be used to convert twenty four Hertz (24 Hz) film to sixty Hertz (60 Hz) video images.
Scan rate conversion techniques may also be used to convert thirty Hertz (30 Hz) high definition (HD) camera images to sixty Hertz (60 Hz) video images.
The additional fields needed for scan rate conversion may be acquired by simply repeating the original fields. The preferred method, however, is to use a progressive to interlace conversion using motion compensated interpolation.
There is a need in the art for an improved method of motion compensation to generate interpolated fields that will provide a sharp video image. In particular, there is a need in the art for an improved method for obtaining accurate values for motion compensated pixels when incorrect motion vectors are assigned to the pixels.
The present invention is designed to be used in a video image upconversion unit of the type that uses motion compensation to generate an interpolated field using motion vectors. The present invention comprises an improved method of motion compensation that performs motion compensation on a pixel and determines whether a motion vector associated with the pixel is correct or incorrect.
If the motion vector is incorrect, the method sets the value of the pixel to a previously recorded pixel value. The previously recorded pixel value is obtained by calculating for the pixel the difference between the value of a corresponding motion compensated pixel from a previous frame and the value of a corresponding motion compensated pixel from a next field. The difference is then compared with a threshold value. If the difference is less than the threshold value, then the value of the pixel is set equal to the average of the value of the corresponding motion compensated pixel from the previous frame and the value of the corresponding motion compensated pixel from the next field.
The improved method of the invention comprises the steps of (1) calculating for a pixel within an array of n pixels by m pixels the difference in value between the value of a corresponding motion compensated pixel from a previous frame and the value of corresponding motion compensated pixel from a next field, (2) comparing the difference with a threshold value, (3) setting the value of the pixel with said array of n pixels by m pixels equal to the average of the value of the corresponding motion compensated pixel from said previous frame and the value of the corresponding motion compensated pixel of the next field if the value of the difference is less than the threshold value, and (4) recording the value for the pixel within the array of n pixels by m pixels.
The improved method of the invention also comprises the steps of (1) calculating for a pixel within a block of pixels the difference in value between the value of a corresponding motion compensated pixel from a previous frame and the value of a corresponding motion compensated pixel from a next field, (2) comparing the difference with a threshold value, and (3) setting the value of the pixel within the block of pixels to the average of the value of the corresponding motion compensated pixel from the previous frame and the value of the corresponding motion compensated pixel of the next field, if the value of the difference is less than the threshold value, and (4) setting the value of the pixel within the block of pixels equal to the value that was previously recorded for the pixel when the pixel was evaluated as a pixel element within the array of n pixels by m pixels, if the value of the difference is not less than the threshold value.
It is a primary object of the present invention to provide an improved method motion compensated upconversion in a video image upconversion unit of the type that uses motion compensation to generate an interpolated video field using motion vectors.
It is an additional object of the present invention to provide a method for obtaining an accurate value for a motion compensated pixel for inclusion in a generated field to be interpolated between a previous frame and a next field.
It is another object of the present invention to provide an improved method for obtaining an accurate value for a motion compensated pixel that will provide a sharp video image in a generated field to be interpolated between a previous frame and a next field by determining whether the motion vector associated with the pixel is correct or incorrect.
It is also an object of the present invention to provide an improved method for obtaining an accurate value for a motion compensated pixel that will provide a sharp video image in a generated field to be interpolated between a previous frame and a next field by providing an accurate pixel value when the motion vector that is associated with the pixel is incorrect.
It is an additional object of the present invention to provide an improved method for determining a threshold value to be used to obtain an accurate value for a motion compensated pixel that will provide a sharp video image in a generated field to be interpolated between a previous frame and a next field.
The foregoing has outlined rather broadly the features and technical advantages of the present invention so that those skilled in the art may better understand the Detailed Description of the Invention that follows. Additional features and advantages of the invention will be described hereinafter that form the subject of the claims of the invention. Those skilled in the art should appreciate that they may readily use the conception and the specific embodiment disclosed as a basis for modifying or designing other structures for carrying out the same purposes of the present invention. Those skilled in the art should also realize that such equivalent constructions do not depart from the spirit and scope of the invention in its broadest form.
Before undertaking the Detailed Description of the Invention, it may be advantageous to set forth definitions of certain words and phrases used throughout this patent document: the terms xe2x80x9cincludexe2x80x9d and xe2x80x9ccomprisexe2x80x9d and derivatives thereof, mean inclusion without limitation; the term xe2x80x9cor,xe2x80x9d is inclusive, meaning and/or; the phrases xe2x80x9cassociated withxe2x80x9d and xe2x80x9cassociated therewith,xe2x80x9d as well as derivatives thereof, may mean to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have, have a property of, or the like; and the term xe2x80x9ccontroller,xe2x80x9d xe2x80x9cprocessor,xe2x80x9d or xe2x80x9capparatusxe2x80x9d means any device, system or part thereof that controls at least one operation, such a device may be implemented in hardware, firmware or software, or some combination of at least two of the same. It should be noted that the functionality associated with any particular controller may be centralized or distributed, whether locally or remotely. Definitions for certain words and phrases are provided throughout this patent document, those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases.